Ground engaging tool wear and loss detection system and method

ABSTRACT

An example wear detection system receives a plurality of images from a plurality of sensors associated with a work machine. Individual sensors of the plurality of sensors have respective fields-of-view different from other sensors of the plurality of sensors. The wear detection system identifies a first region of interest and second region of interest associated with the at least one GET. The wear detection system determines a first set of image points and a second set of images points for the at least one GET based on geometric parameters associated with the GET. The wear detection system determines a wear level or loss for the at least one GET based on the GET measurement.

TECHNICAL FIELD

The present disclosure relates to a system and method for detecting wearof objects over time, and more particularly, to a system and method fordetecting wear in, or loss of, one or more ground engaging tools (GET)over time using imaging techniques.

BACKGROUND

Machines may be used to perform variety of tasks at a worksite. Forexample, machines may be used to excavate, move, shape, contour, and/orremove material present at the worksite, such as gravel, concrete,asphalt, soil, and/or other materials. These machines can include abucket used to collect such materials, and the bucket can include a setof GET, such as teeth, to loosen the material. GET can also includeshrouds attached to the bucket between teeth to protect the edge of thebucket. Over time, the GET wear and diminish in size reducing theireffectiveness making it more difficult for the bucket to collectworksite material. GET can also break from the bucket. When a GET breakgoes undetected, the GET can mix with the worksite material and cancause damage to downstream processing equipment such as crushers orpulverizers. Work machines may utilize wear detection systems toidentify worn or broken GET before damage to downstream equipmentoccurs.

An attempt to provide a wear detection system is described in WIPO Pub.No. WO2020237324A1 (“the '324 Publication”), published Dec. 2, 2020. The'324 Publication describes a system having one or more sensors mountedon working equipment, directed toward a GET to monitor status of theGET. The system receives data from the one or more sensors and generatesa three-dimensional (3D) representation of at least a portion of the GETand compares the currently generated 3D representation of the GET with apreviously generated 3D representation of the GET. The system uses thecomparison to determine wear or loss of the GET.

However, sole reliance on 3D representations for detecting GET wear orloss, as described in the '324 Publication, has disadvantages. 3Drepresentations of GET can be overly dense and processing them canconsume considerable computing resources. In environments with manysensors, requiring multiple scans of GET during the dig-dump cycle ofthe work machine, or where speed of GET wear detection is critical,processing 3D representations is not feasible due to computationalcomplexity. As a result, the system described in the '324 Publicationmay not be suitable for some environments. The systems and methodsdescribed herein are directed to addressing one or more of theseconcerns.

SUMMARY

According to a first aspect, a GET wear detection method includesreceiving a plurality of images from a plurality of sensors associatedwith a work machine, the plurality of images comprising images of atleast one ground engaging tool (GET) of the work machine, whereinindividual sensors of the plurality of sensors have respectivefields-of-view different from other sensors of the plurality of sensors.The method also includes identifying a first region of interestassociated with the at least one GET and included in a first image ofthe plurality of images, the first region of interest including firstdata characterizing the at least one GET. The method further includesidentifying a second region of interest associated with the at least oneGET and included in a second image of the plurality of images, thesecond region of interest including second data characterizing the atleast one GET. The method determines a first set of image points for theat least one GET, the first set of image points being determined basedat least in part on geometric parameters associated with the at leastone GET and first edges of the at least one GET detected in the firstdata. The method also determines a second set of image points for the atleast one GET, the second set of image points being determined based atleast in part on the geometric parameters associated with the at leastone GET and second edges of the at least one GET detected in the seconddata. The method also includes determining a GET measurement for the atleast one GET based at least in part on the first set of image pointsand the second set of image points, and determining a wear level or lossfor the at least one GET based on the GET measurement.

According to a further aspect, a GET wear detection system includes oneor more processors and a plurality of sensors associated with a workmachine. The plurality of sensors have differing fields-of-view directedto capture a plurality of images of at least one GET of the workmachine. The GET wear detection system also includes a non-transitorycomputer readable media storing executable instructions that whenexecuted by the one or more processors cause the one or more processorsto perform operations including identifying a first region of interestassociated with the at least one GET and included in a first image ofthe plurality of images, the first region of interest including firstdata characterizing the at least one GET. The operations also includeidentifying a second region of interest associated with the at least oneGET and included in a second image of the plurality of images, thesecond region of interest including second data characterizing the atleast one GET. The operations further include determining a first set ofimage points for the at least one GET, the first set of image pointsbeing determined based at least in part on geometric parametersassociated with the at least one GET and first edges of the at least oneGET detected in the first data. The operations further includedetermining a second set of image points for the at least one GET, thesecond set of image points being determined based at least in part onthe geometric parameters associated with the at least one GET and secondedges of the at least one GET detected in the second data. Theoperations further include determining a GET measurement for the atleast one GET based at least in part on the first set of image pointsand the second set of image points, and determining a wear level or lossfor the at least one GET based on the GET measurement.

According to another aspect, a work machine includes a bucket comprisingat least one GET, a stereoscopic camera comprising a left image sensorand a right image sensor, an infrared sensor, one or more processors,and non-transitory computer readable media storing executableinstructions that when executed by the one or more processors cause theone or more processors to perform operations. The operations includereceiving a left image of the at least one GET captured by the leftimage sensor, receiving a right image of the at least one GET capturedby the right image sensor, and generating a disparity map based on theleft image and the right image. The operations further includeidentifying a first region of interest associated with the at least oneGET based on the disparity map. The operations also include receivinginfrared image data from the infrared sensor and identifying a secondregion of interest associated with the at least one GET within theinfrared image data. The operations determine a first set of imagepoints for the at least one GET, the first set of image points based atleast in part on geometric parameters associated with the at least oneGET and first edges of the at least one GET detected in the disparitymap and determine a second set of image points for the at least one GET,the second set of image points determined based at least in part on thegeometric parameters associated with the at least one GET and secondedges of the at least one GET detected in the second data. Theoperations further determine a GET measurement for the at least one GETbased at least in part on the first set of image points and the secondset of image points and determine a wear level or loss for the at leastone GET based on the GET measurement.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit of a reference numberidentifies the figure in which the reference number first appears. Thesame reference numbers in different figures indicate similar oridentical items.

FIG. 1 is a block diagram depicting a schematic view of an examplemachine including an example system for detecting wear in GET.

FIG. 2 is a diagram depicting a schematic side view of exampleenvironment having an example machine including an example system fordetecting wear in GET.

FIG. 3 is a diagram depicting a schematic side view of another exampleenvironment having an example machine including an example system fordetecting wear in GET.

FIG. 4 is a diagram depicting a schematic side view of another exampleenvironment having an example machine including an example system fordetecting wear in GET

FIG. 5 is an image data flow diagram depicting an example flow of imagedata for a region of interest detection process using computer visiontechniques.

FIG. 6 is an image data flow diagram depicting an example flow of imagedata for a region of interest detection process using deep learningtechniques.

FIG. 7 is an image data flow diagram depicting an example flow of imagedata for a region of interest detection process using deep learningtechniques.

FIG. 8 is an image data flow diagram depicting an example flow of imagedata for a region of interest detection process using a LiDAR sensor andimaging data.

FIG. 9 is an image data flow diagram depicting an example flow of imagedata for a wear detection process using image points.

FIG. 10 is an example process for detecting wear in an exampleenvironment.

FIG. 11 is an example process for detecting wear in an exampleenvironment.

FIG. 12 is an example user interface of an example system for detectingwear in GET.

DETAILED DESCRIPTION

The present disclosure is generally directed to systems and methods fordetecting wear of components of a work machine in an environment, suchas a worksite, using one or more sensors. The one or more sensors caninclude imaging sensors (that could be part of a stereoscopic camera or“stereo camera”), LiDAR sensors, infrared (IR) sensors, sonar sensors,temperature sensors, or radar sensors capable of capturing imaging dataassociated with the components. The imaging data can include, but is notlimited to, video, images, LiDAR imaging data, IR imaging data,sound-based imaging data, or radar data. The imaging data is analyzed bya wear detection computer system associated with the workmachine—aspects of which may be disposed on the work machine, within thestereo camera, within the one or more sensors of the work machine, orexternal to these or external to the work machine—to detect wear of thecomponent. The component can be one or more GET of a bucket of the workmachine, as one example. The one or more sensors of the work machine mayeach have differing fields-of-view and may produce slightly differentimaging data for the component. The differing fields-of-view can reduceerrors related to poor lighting conditions, shadows, or debris thatcould negatively affect imaging of the components. The wear detectionsystem can determine image points which may relate to marker points onGET such as edges, corners, or visual indicators on the components—fromcaptured imaging data and use the image points to determine measurementsof the components. Based on the determined measurements, and/orhistorical or baseline measurements, the wear detection system candetermine a wear level or loss of the component. The wear detectionsystem may receive imaging data from the one or more sensors at varioustime instances in the dig-dump cycle of the work machine, and maydetermine whether to generate an alert or capture additional imagingdata based on the placement of the time instance within the dig-dumpcycle.

FIG. 1 is a block diagram depicting a schematic of an example workmachine 100 including an example wear detection computer system 110.While FIG. 1 depicts work machine 100 as a hydraulic mining shovel, inother examples, work machine 100 can include any machine that moves,sculpts, digs, or removes material such as soil, rock, or minerals. Asshown in FIG. 1 , work machine 100 can include a bucket 120 attached toarm 122. Bucket 120 can include one or more ground engaging tools (GET)125, such as teeth, that assist work machine 100 in loosening material.While the examples provided in this disclosure typically refer to GET125 as teeth, other types of GET are contemplated to be within the scopeof the embodiments provided by this disclosure. For example, GET caninclude lip shrouds, edge guards, adapters, ripper protectors, cuttingedges, sidebar protectors, tips, or any other tool associated with awork machine that wear over time due to friction with worksite material.

Work machine 100 also includes one or more sensors having respectivefields of view such as sensor 126 having field-of-view 127 and stereocamera 128 having field-of-view 129. Both field-of-view 127 andfield-of-view 129 are directed to bucket 120 and GET 125. As shown inFIG. 1 , field-of-view 127 and field-of-view 129 overlap but differ.Sensor 126 can include image sensors, LiDAR sensors, IR sensors, sonarsensors, or radar sensors as just some examples.

As used in the in the present disclosure, the term “imaging data” refersto data produced by sensor 126 or stereo camera 128 and received by weardetection computer system 110 that can be interpreted or processed toreflect the size, shape, or appearance of GET 125. While the presentdisclosure refers to sensor 126 in the singular, in some embodiments,work machine 100 will typically include more than one sensor 126 inaddition to stereo camera 128, each with their own respectivefield-of-view 127. For example, work machine 100 may include camera 128having field-of-view 129, a LiDAR sensor, an additional imaging sensor,and an IR sensor, all of which may produce imaging data processed bywear detection computer system according to disclosed embodiments.

In some embodiments, sensor 126 comprises an adaptive scanning LiDARsensor, i.e., a LiDAR sensor for which its resolution and field of viewcan be commanded, controlled, and configured. For example, sensor 126can include an AEYE 4Sight M™. In some embodiments, field-of-view 127starts with a baseline of 60 degrees by 30 degrees (representing a “low”resolution range scan) which can then be adjusted by 0.1 degrees to highdefinition region of interest spanning 0.025 degrees, but other fieldsof view and angular resolutions may be present in other embodiments.Sensor 126 can be configured to collect as many as 1,600 points persquare degree at a frequency of 100 Hz. The precision of sensor 126 is afunction of the angular resolution of field-of-view 127 and the distancebetween sensor 126 and GET 125. As an example, when GET 125 isapproximately six meters from sensor 126 and field-of-view 127 isconfigured as 60 degrees by 30 degrees, a 1,600 points-per-square degreescan would yield LiDAR hits within an captured rectangle ofapproximately 7.2 meters by 3.2 meters. By refocusing the field of view,a LiDAR hit can register 2.6 millimeters in the horizontal and verticaldirections. While the above describes one example sensor 126, differentLiDAR sensors capable of adaptive scanning can be used in variousembodiments.

Sensor 126 can also include an infrared sensor or a sensor capable ofdetecting a heat signature of GET 125. For example, sensor 126 caninclude a long-wave, FLIR® infrared camera with a 640×512 resolution, 9Hz refresh rate, and a 75-degree field of view. An infrared sensor 126can complement camera 128 in environments where there is low light orwhere debris may stick to GET 125 during operation. Other examples ofsensor 126 can include sonar or radar sensors.

Stereo camera 128 includes a left image sensor and a right image sensorthat are spaced apart as to capture a stereo image of objects withinfield-of-view 129, such as bucket 120 and GET 125. In some embodiments,the left image sensor and the right image sensor capture monochromaticimages. Stereo camera 128 can also include a color image sensor tocapture color images of objects within field-of-view 129. In someembodiments, camera 128 outputs digital images or work machine 100 mayinclude an analog to digital converter disposed between camera 128 andwear detection computer system 110 to covert analog images to digitalimages before they are received by wear detection computer system 110.While the present disclosure refers to stereo camera 128 having a singlefield-of-view 129 for ease of discussion, those having skill in the artwill understand that each image sensor (e.g., left, right, color) ofcamera 128 has its own respective field-of-view to generate stereoscopicimages from which GET 125 can be measured according to disclosedembodiments.

The one or more sensors of work machine 100, such as sensor 126 andcamera 128, can include a lens cleaning device to remove debris, fog, orother obstructions from the surface (or screen) of the lenses of the oneor more sensors in some embodiments. The lens cleaning device caninclude, for example, a nozzle for emitting compressed air, washersolvent, or washer antifreeze solvent. The lens cleaning device can alsoinclude a moving wiper that is configured to contact and wipe thesurface of the lens to push debris or other obstructions away from thelens surface. In some embodiments, the cover of the lenses of the one ormore sensors may include an actuator that rotates the lens screen (forcylindrical lens screens) or slides the lens screen (for flat lensscreens) so that it contacts one or more wipers to remove debris fromthe screen.

As work machine 100 operates within a worksite, it may move arm 122 toposition bucket 120 to move or dig material within the worksite as partof a dig-dump cycle. As work machine 100 positions bucket 120 throughthe dig-dump cycle, bucket 120 may move in and out of field-of-view 127and field-of-view 129. Sensor 126 and camera 128 may be positioned sothat they have an unobstructed view of GET 125 during the dig-dumpcycle. For example, sensor 126 and camera 128 may be positioned on workmachine 100 so that bucket 120 and GET 125 are visible at the momentbucket 120 empties material within the dig-dump cycle. As anotherexample, sensor 126 and camera 128 may be positioned so that bucket 120enters their fields-of-view when arm 122 is fully extended or fullycontracted within the dig-dump cycle. As explained below with respect toFIGS. 2-4 , the position of sensor 126 and camera 128 (and accordinglyfield-of-view 127 and field-of-view 129) may vary depending on the typeof work machine 100 and specifics related to its worksite.

In some embodiments, field-of-view 127 and field-of-view 129 may captureimage data of bucket 120 and GET 125 at different points in the dig-dumpcycle. For example, sensor 126 may capture image data of GET 125 at anearly part of the dig-dump cycle (e.g., closer to the start of the cyclethan the end of the cycle), and camera 128 may capture image data of GET125 in a late part of the dig-dump cycle (e.g., closer to the end of thecycle than the start of the cycle). In some embodiments, sensor 126and/or camera 128 may adjust their respective fields-of-view 127, 129 tocollect image data of GET 125 at different points in the dig-dump cycle.For example, in some embodiments, both sensor 126 and camera 128 maycapture image data of GET 125 at an early part of the dig-dump cycle,then adjust fields-of-view 127, 129 to capture image date of GET 125 ina late part of the dig-dump cycle.

According to some embodiments, work machine 100 includes an operatorcontrol panel 130. Operator control panel 130 can include a display 133which produces output for an operator of work machine 100 so that theoperator can receive status or alarms related to wear detection computersystem 110. Display 133 can include a liquid crystal display (LCD), alight emitting diode display (LED), cathode ray tube (CRT) display, orother type of display known in the art. In some examples, display 133includes audio output such as speakers or ports for headphones orperipheral speakers. Display 133 can also include audio input devicessuch as microphone or ports for peripheral microphones. Display 133includes a touch-sensitive display screen in some embodiments, whichalso acts as an input device.

Display 133 can display information about the wear level or loss of GET125 that has been rendered by wear computer detection system 110. Forexample, display 133 may display calculated measurements of GET 125. Thecalculated measurements may be color coded in some embodiments toreflect a health status of GET 125. For example, the calculatedmeasurements may be displayed with a green background if GET 125 isconsidered to have an acceptable wear level, yellow background if GET125 are close to needing replacement, or red if GET 125 have broken orworn to the point of needing replacement. Display 133 can also show animage of bucket 120, GET 125, or an image of a region of interest withinfields-of-view 127, 129 related to GET 125 rendered by wear computerdetection system 110.

In some embodiments, operator control panel 130 also includes a keyboard137. Keyboard 137 provides input capability to wear detection computersystem 110. Keyboard 137 includes a plurality of keys allowing theoperator of work machine 100 to provide input to wear detection computersystem 110. For example, an operator may depress the keys of keyboard137 to select or enter the type of work machine 100, bucket 120, and/orGET 125 according to examples of the present disclosure. Keyboard 137can be non-virtual (e.g., containing physically depressible keys) orkeyboard 137 can be a virtual keyboard shown on a touch-sensitiveembodiment of display 133.

As shown in FIG. 1 , wear detection computer system 110 includes a oneor more processors 140. Processor(s) 140 can include one or more of acentral processing unit (CPU), a graphics processing unit (GPU), afield-programmable gate array (FPGA), some combination of CPU, GPU, orFPGA, or any other type of processing unit. Processor(s) 140 may havenumerous arithmetic logic units (ALUs) that perform arithmetic andlogical operations, as well as one or more control units (CUs) thatextract instructions and stored content from processor cache memory, andthen executes the instructions by calling on the ALUs, as necessary,during program execution. Processor(s) 140 may also be responsible forexecuting drivers and other computer-executable instructions forapplications, routines, or processes stored in memory 143, which can beassociated with common types of volatile (RAM) and/or nonvolatile (ROM)memory.

Wear detection computer system 110 also includes a memory 143. Memory143 can include system memory, which may be volatile (such as RAM),non-volatile (such as ROM, flash memory, etc.) or some combination ofthe two. Memory 143 can further include non-transitory computer-readablemedia, such as volatile and nonvolatile, removable and non-removablemedia implemented in any method or technology for storage ofinformation, such as computer-readable instructions, data structures,program modules, or other data. System memory, removable storage, andnon-removable storage are all examples of non-transitorycomputer-readable media. Examples of non-transitory computer-readablemedia include, but are not limited to, RAM, ROM, EEPROM, flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other non-transitorymedium which can be used to store the desired information and which canbe accessed by wear detection computer system 110.

Memory 143 stores data, including computer-executable instructions, forwear detection computer system 110 as described herein. For example,memory 143 can store one or more components of wear detection computersystem 110 such as a physical parameter library 145, an image analyzer150, a wear analyzer 153, an alert manager 155, and GET wear levelstorage 157. Memory 143 can also store additional components, modules,or other code executable by processor(s) 140 to enable operation of weardetection computer system 110. For example, memory 143 can include coderelated to input/output functions, software drivers, operating systems,or other components.

According to some embodiments, aspects of wear detection computer system110 may be disposed within camera 128. For example, camera 128 mayinclude one or more of processor(s) 140 and/or memory 143. Similarly,aspects of wear detection computer system 110 may be disposed withinsensor 126. In addition, or alternatively, aspects of wear detectioncomputer system 110 may be disposed on work machine 100 and outside ofsensor 126 or camera 128.

Physical parameter library 145 includes physical parameter sets relatedto work machine 100, bucket 120, GET 125, sensor 126 and/or camera 128.For example, physical parameter library 145 can include measurement datarelated to the size of bucket 120, shape of bucket 120, size of GET 125,shape of GET 125, and the spatial relationship between GET 125 andbucket 120, and/or the spatial relationship between sensor 126 andcamera 128, as just some examples. Physical parameter library 145 canalso include parameters related to the size and shape of GET 125 in anew or unworn state and parameters related to the size and shape of GET125 when they have reached maximum wear.

Physical parameter library 145, in some embodiments, may includegeometric parameters related to marker points of bucket 120 or GET 125.The marker points are related to aspects or reference points of bucket120 and GET 125 that can be used by wear analyzer 153 to measure wear orloss of GET 125. For example, the marker points can include corners ofGET 125, the edge of bucket 120 where GET 125 engage with bucket 120,the corner between an edge of bucket 120 and GET 125, or physicalmarkers applied to GET 125 such as welds, paint, grooves, reflectivetape, or bar codes. Physical parameter library 145 may include physicalparameter sets having information about the relative location of themarker points to GET 125 or bucket 120. For example, the physicalparameter sets can include angles of the corners of GET 125, therelative location of physical markers on GET 125 to the edge of GET 125or the edge of bucket 120. Embodiments of wear detection computer system110 may include physical parameter sets beyond the specific examplesdescribed here, and a person of ordinary skill in the art wouldunderstand that other methods of detecting the marker points on bucket120 or GET 125 can be used and that physical parameter library 145 mayinclude physical parameters sets that can be used by wear analyzer 153to identify the marker points in image data collected by sensor 126and/or camera 128.

Physical parameter library 145 can also include templates or referenceimages related to the combination of bucket 120 and GET 125 (e.g., abucket-tool template). For example, for work machine 100, one of thetemplates stored in physical parameter library 145 can include an imageof bucket 120 with GET 125 as bucket 120 is expected to be positionedwithin field-of-view 127 and/or field-of-view 129. The bucket-tooltemplates can represent GET 125 that are unworn (e.g., unworn orexpected edges) or GET 125 that have reached maximum wear (e.g., athreshold edge). Physical parameter library 145 can also include otherinformation related to the wear of GET 125 to assist wear analyzer 153in determining when GET have worn to the point of needing replacement.Wear data related to GET 125 can be in the form of actual measurement(e.g., metric or imperial dimensions) or in the form of pixel values, asjust some examples.

As another example, physical parameter library 145 can include CAD-basedmodels of GET 125. The CAD-based models can be models of reference GET125 developed using computer-aided design programs such as AutoCAD®,Autodesk®, SolidWorks®, or other well-known CAD program. The CAD-basedmodels can be used by wear detection computer system 110 as referencepoints to compare observed GET 125 sizes and shapes to a model,standard, or unworn GET of the same type to determine wear or loss ofGET 125. In some embodiments, the CAD-based models may include thelocation, orientation, and/or relative positioning of the points on GET125.

Physical parameter library 145 can include multiple physical parametersets where each physical parameter set corresponds to a work machine,bucket, GET, or a combination of these. During operation, an operatormay use operator control panel 130 to select a physical parameter setfrom physical parameter library 145 matching bucket 120 and GET 125, orwork machine 100. For example, if the work machine 100 is a hydraulicmining shovel having a model number “6015B,” the operator may useoperator control panel 130 to input the model number “6015B,” and weardetection computer system 110 may load into memory 143 a physicalparameter set corresponding to a model 6015B hydraulic mining shovelfrom physical parameter library 145. In some examples, a list oftemplates available in physical parameter library 145 can be shown ondisplay 133 upon a power-up or reset operation of wear detectioncomputer system 110, and an operator may select one of the physicalparameter sets from the list for operation depending on the model numberof work machine 100, bucket type of bucket 120, or type of GET 125.

In some embodiments, the operator may position bucket 120 and GET 125within field-of-view 129 of camera 128 at the beginning of a work shiftand cause wear detection computer system 110 to capture an image ofbucket 120 and GET 125 using an input on operator control panel 130.Wear detection computer system 110 may then perform an image matchingprocess to match bucket 120 and GET 125 with a physical parameter setand configure itself for the wear detection and image processingprocesses disclosed herein based on the matching physical parameter set.In some embodiments, wear detection computer system 110 may use sensor126 and field-of-view 127 for this configuration process instead ofcamera 128 and field-of-view 129.

Image analyzer 150 can be configured to analyze imaging data captured byeither sensor 126 or camera 128 to identify GET 125 within field-of-view127 and field-of-view 129 and to measure wear of GET 125 based onprocessing of that imaging data. For example, image analyzer 150 canreceive stereoscopic images from camera 128 in the form of leftrectified images (captured by the left image sensor of camera 128) and aright rectified image (captured by the right image sensor of camera128). Image analyzer 150 may perform various computer vision techniqueson the left rectified image and the right rectified image to identify ordetermine a region of interest corresponding to GET 125. As anotherexample, image analyzer 150 may receive imaging data captured by sensor126 which can be used to identify a region of interest corresponding toGET 125. In the disclosed embodiments, image analyzer 150 receives datafrom sensor 126 to determine wear or loss of GET 125, as described inmore detail below.

In some embodiments, image analyzer 150 processes two sets of theimaging data when detecting wear or loss of GET 125. The first set ofimaging data is captured to identify a region of interest withinfield-of-view 127 or field-of-view 129. The region of interestcorresponds to the relative location of GET 125 within field-of-view 127or field-of-view 129. The first set of imaging data—for detecting theregion of interest—is a broad and lower resolution imaging data captureintended to locate a general region of interest for GET 125 and may bereferred to as a “coarse scan.” In some embodiments, the first set ofimaging data can be captured using camera 128, and image analyzer 150determines the region of interest using computer vision or machinelearning techniques. In other embodiments, the first set of imaging datacan be captured using sensor 126 at a first, lower resolution (e.g., 60degrees by 30 degrees in a LiDAR embodiment of sensor 126) that isrelatively wide. In some implementations, image analyzer 150 receivesthe first set of imaging data from both sensor 126 and camera 128.

When image analyzer 150 identifies a region of interest corresponding toGET 125, it may then control sensor 126 to focus on the specific regionof interest to perform a higher-resolution scan, or “fine scan” in someembodiments. For example, image analyzer 150 may communicate with theapplication programming interface (API) of sensor 126 to command it tochange field-of-view 127 to become narrower with a focus on theidentified region of interest. Sensor 126 then performs another scan ofGET 125 to collect a second set of imaging data. The second set ofimaging data—having been captured by sensor 126 with a narrowerfield-of-view 127—will be of higher resolution than the first imagingdata captured by either sensor 126 (when set with a wide field of view)or camera 128.

In some embodiments, the coarse scan can be performed by either sensor126 or camera 128 and the fine scan can be performed by the other ofcamera 128 or sensor 126. Alternatively, both sensor 126 and camera 128may perform both the coarse scan and the fine scan.

In one embodiment, image analyzer 150 creates a dense stereo disparitymap based on the left rectified image and the right rectified imagereceived from camera 128. Image analyzer may segment the dense stereodisparity map to identify the region of interest. In addition, imageanalyzer 150 may also create a three-dimensional point cloud based onthe dense stereo disparity map and may segment the three-dimensionalpoint cloud to identify the region of interest.

In addition to computer vision techniques, or as an alternative to usingcomputer vision techniques, image analyzer 150 can also employ deeplearning or machine learning techniques to identify regions of interestwithin left rectified images and/or right rectified images captured bycamera 128. For example, image analyzer 150 may use a deep learning GETdetection algorithm that employs a neural network that has been trainedto identify regions of interest based on a corpus of images whereindividual GET, groups of GET, or GET and bucket combinations have beenlabeled. Image analyzer 150 may also use a deep learning GET-locationalgorithm that employs a neural network that has been trained to locateGET within an image. The GET-location algorithm can be trained usingcorpus of images where individual GET have been labeled. Once theGET-location algorithm identifies individual GET within an image, itoutputs the corresponding location for the GET. For example, theGET-location algorithm can output a pixel location or a bounding boxoutput related to the location of the GET.

In some embodiments, the deep learning GET detection algorithm includesa neural network that has been trained to identify regions of interestbased on disparity maps. For example, the corpus of training data forthe deep learning GET detection algorithm may include disparity imagesbetween a left-rectified and right-rectified image where individual GET,groups of GET, or GET and bucket combinations have been labeled.

As noted above, once image analyzer 150 identifies the region ofinterest including GET 125, it may command and control sensor 126 tofocus field-of-view 127, or command and control camera 128 to focusfield-of-view 129 on the region of interest. In some embodiments, imageanalyzer 150 uses spatial relationship data between sensor 126 andcamera 128 to command sensor 126 to alter field-of-view 127 on theregion of interest. For example, in LiDAR embodiments, once sensor 126receives commands to change its field-of-view, it may alter theconfiguration of its MEMS (micro-electromechanical system) mirrors tonarrow field-of-view 127 to capture higher-resolution imaging datarelated to GET 125.

From the captured higher-resolution imaging data, image analyzer 150 cancreate a three-dimensional point cloud corresponding to GET 125. Eachpoint in the three-dimensional point cloud corresponds to a “hit” ordetection point captured by sensor 126, such as a LiDAR hit or aninfrared heat signature. In some embodiments, the real-life distancebetween the points can be as small as 1 millimeter. In embodiments withsufficiently high resolution (i.e., where the real-life distance betweenpoints is less than approximately 2.5 mm), image analyzer 150communicates the three-dimensional point cloud data to wear analyzer 153for wear detection analysis. In other embodiments, image analyzer 150may perform additional processing of the three-dimensional point clouddata to further refine it for wear analysis.

For example, in some embodiments, image analyzer 150 converts thethree-dimensional point cloud to a dense mesh surface. Image analyzer150 may further convert the dense mesh surface to a sparse mesh surfacebefore communicating the GET imaging data to wear analyzer 153.Conversion from a three-dimensional point cloud, to a dense meshsurface, then to a sparse mesh surface may be desirable to reducecomputational expenditure when comparing the imaging data captured bysensor 126 to a CAD-based GET model. Conversion from a three-dimensionalpoint cloud, to a dense mesh surface, then to a sparse mesh surface canalso filter out noise that may be present in the imaging data due tooversampling.

In some embodiments, wear analyzer 153 fuses the lower-resolution,first-received imaging data from camera 128 with the higher-resolutiondata, second-received imaging data received from sensor 126 to gainconfidence in the observed measurement of GET 125. In such embodiments,image analyzer 150 performs additional processing on the left image andright image captured by camera 128. For example, once image analyzer 150identifies the regions of interest it can further process them to createa left-edge digital image corresponding to the left rectified image anda right-edge digital image corresponding to the right rectified image.Image analyzer 150 may employ gradient magnitude search-based edgedetection, but other edge detection techniques employed within the fieldof computer vision (e.g., zero-crossing based edge detection techniques)could be employed in other embodiments to create the left-edge digitalimage and the right-edge digital image.

In some examples, image analyzer 150 may refine edge estimates of GET125 and/or identify individual GET 125 by using an expected location ofGET 125 within the captured image. For example, image analyzer 150 mayknow the expected position of GET 125 relative to bucket 120 based onthe physical parameter set stored in physical parameter library 145corresponding to the type of bucket 120 and GET 125 in use. Using thisinformation, image analyzer 150 can go to the expected location inselected image and capture a pixel region proximate to the teeth. Thepixel region can then be used to further identify the tooth based oncomputer vision techniques such as application of a convolution filter,segmentation analysis, edge detection, or pixel strength/darknessanalysis within the pixel region. In some embodiments, image analyzer150 may use an individual tooth template to apply to the pixel region tofurther refine the location of the tooth using computer visiontechniques. Image analyzer 150 may further refine edges using dynamicprogramming techniques. Dynamic programming techniques can includesmoothing based on the strength of the edge, whether the edge is closeto a hole or region of uncertainty in the dense stereo disparity map, orother edge detection optimization techniques. Image analyzer 150 canalso use the output of the GET-location algorithm to gain confidence inthe determining the location of the GET and to further refine edgeestimates based on the output of the GET-location algorithm.

Image analyzer 150 may also create a sparse stereo disparity that isprovided to wear analyzer 153 that wear analyzer 153 can use along withthe higher-resolution imaging data captured by sensor 126 to determinewear or loss in GET 125. In some embodiments, image analyzer 150 createsthe sparse stereo disparity between the left-edge digital image(associated with the left rectified image) and the right-edge digitalimage (associated with the right rectified image), and this disparity isused by wear analyzer 153. Alternatively, the sparse stereo disparitymay be calculated from a first region of interest image (associated withthe left rectified image) and a second region of interest image(associated with the right rectified image) and image analyzer 150 maydetect an edge from the sparse stereo disparity image.

Wear analyzer 153 can be configured to analyze the sparse stereodisparity generated by image analyzer 150 for wear. For example, thephysical parameter set associated with bucket 120 and GET 125 caninclude expected data related to unworn GET 125 or a set of unworn GET125 that has been calibrated based on the expected image capture ofcamera 128. The expected data can be in the form of pixels, actualmeasurement, a CAD-based model of GET 125 or an edge image related tounworn GET, as just some examples. Once wear analyzer 153 receives thesparse stereo disparity, it can fuse and correlate that sparse stereodisparity with the three-dimensional point cloud) of thehigher-resolution imaging data captured by sensor 126 (or, in someembodiments, the dense mesh surface or sparse mesh surface determinedbased on the three-dimensional point cloud) to determine measurementdata related to the GET 125. It may then compare the determinedmeasurement data to expected data corresponding to an unworn version ofGET 125 to determine wear levels, or loss, for GET 125.

In some embodiments, image analyzer 150 identifies marker points inimage data collected from sensor 126 and/or camera 128. Marker points,as noted above, can refer to corners of GET 125, edges of bucket 120that meet GET 125, or can refer to physical or visual markers on bucket120 or GET 125. For example, when image analyzer 150 is identifyingcorners of GET 125 as marker points it may perform corner detectiontechniques known in the field of computer vision such as Moravec, Harris& Stephens, Shi-Tomasi, Forstner, multi-scale Harris, Laplacian ofGaussian, Wang and Brady, SUSAN (smallest univalue segment assimilatingnucleus), Trajkovic and Hedley and/or Hessian techniques as just someexamples. Other methods of corner detection be used, as well asperforming segmentation, pattern matching, or template matching analysisbased on image data collected by sensor 126 and/or camera 128 toidentify marker points.

In some embodiments, in addition to, or as an alternative to, computervision techniques, image analyzer 150 may use deep learning or machinelearning techniques to identify marker points within regions ofinterest. For example, image analyzer 150 may deploy a marker pointdetection algorithm that has been trained using a corpus of data wheremarker points (e.g., corners, edges, markers) of GET 125 have beenlabeled. The marker point detection algorithm can be trained usingrectified, monochrome images (e.g., similar to imaging data provided bythe left-image and right-image sensors of camera 128), color images(e.g., similar to imaging data provided by color image sensor of camera128), disparity maps (e.g., similar to a disparity generated based onimaging data provided by the left-image and right-image sensors ofcamera 128), LiDAR point cloud imaging data and/or infrared imagingdata, as just some examples. Training data for marker point detectionalgorithm may vary, and correspond with, the embodiment of sensor 126and camera 128 deployed in embodiment of work machine 100.

Image analyzer 150 also maps image points in the imaging data acrossimage data sources. For example, image analyzer 150 may determine imagepoints related to the end tips of GET 125 in imaging data captured bysensor 126 and imaging data captured by 128. Image analyzer 150 may dothis to determine errors in imaging data that might be caused by debrisor poor lighting conditions. For example, image analyzer 150 maydisregard image point data from one set of imaging data captured bysensor 126 that appears to be an outlier when compared to image pointdata of another set of imaging data captured by the imaging data ofcamera 128, or vice versa. In some embodiments, image analyzer 150 maycompare image point data determined from recently captured imaging datato historical image point data and disregard image point data that isinconsistent with the historical image point data.

In some embodiments, pixel counts associated with the sparse stereodisparity can be used to measure the wear or loss of GET. Pixel countscan include area (e.g., total pixel for the GET), height of the GET inpixels, width of the GET in pixels, the sum of height and width of theGET, as just some examples. The manner of determining pixel counts canvary depending on the shape and style of the GET. For example, for GETthat are much longer than they are wide, height pixel counts may beused, whereas for GET that are much wider than they are long, widthpixel counts may be used. Various methods for determining pixel countsmay be used without departing from the spirit and scope of the presentdisclosure.

In some embodiments, wear analyzer 153 can calculate a similarity scorebetween the determined measurement data and the expected datacorresponding to unworn GET 125. The similarity score can reflect ameasure of how well the determined measurement data of GET 125 matchesthe expected data of the physical parameter set. For example, thesimilarity score can include use of an intersection of union or JaccardIndex method of detecting similarity. In some embodiments, a dicecoefficient or F1 Score method of detecting similarity can be employedto determine the similarity score. The similarity score can also includea value reflecting a percentage of how many pixels of the sparse stereodisparity overlap with the expected edge image. In some embodiments, thesimilarity score may be scaled or normalized from zero to one hundred.

The similarity score can provide an indication of wear of GET 125. Forexample, a low score (e.g., a range of 0 to 20) may indicate that one ofGET 125 has broken or is missing indicating tooth loss. A high score(e.g., a range 80-100) may indicate that a tooth is in good health andneeds no replacing. A score in between the low and high scores canprovide a wear level for the tooth, with higher scores indicating alonger lead time for tooth replacement than a lower score.

In some embodiments, wear analyzer 153 can collect measurement datarelated to GET 125 over time and use the collected measurement data todetermine a wear level of GET 125 and a wear trend of GET 125. Wearanalyzer 153 may store collected measurement data in GET wear levelstorage 157. For example, work machine 100 can be operating in itsworksite over several days for a job. As work machine 100 moves materialduring the job, camera 128 provides stereo images bucket 120 and GET 125to wear detection computer system 110, and image analyzer 150 createssparse stereo disparities for GET 125. Wear analyzer 153 can mapmeasurement data (e.g., pixel counts, metric measurements, imperialmeasurements) associated with the GET 125 at several instances of timeover the period of time of the job. As bucket 120 and GET 125 engagewith material at the worksite, it is expected that GET 125 will diminishin size due to wear. Accordingly, the measurement data associated withGET 125 will likewise decrease over time, and the pixel counts over timewill reflect a wear trend. Wear analyzer 153 can determine a wear levelfor GET 125 at a particular point in time using the wear trend at theparticular point in time. The wear level for GET 125 may indicate thatGET 125 need replacement or it may indicate loss of one or more of GET125. In some embodiments, measurement data associated with GET 125 canbe stored in memory 143 and applied to multiple jobs and multipleworksites, and the wear trend can be applicable to the lifetime of GET125. In such embodiments, pixel counts associated with GET 125 capturedby wear analyzer 153 may be reset when bucket 120 or GET 125 arereplaced, and wear analyzer 153 can restart collection of pixel countsfor GET 125 from a zero-time point.

Since wear analyzer 153 determines a wear trend based on measurementdata for GET 125 measured over time, wear analyzer 153 can also formpredictions of when GET 125 may need replacement. For example, if wearanalyzer 153 determines that measurement data associated with GET 125show that GET 125 lose 1% of life per ten work hours (because themeasurement data decreases by 1% per ten work hours), and GET 125 havebeen used for eight hundred work hours, wear analyzer 153 may determinethat GET 125 need to be replaced within 200 hours.

In some embodiments, wear detection computer system 110 can includealert manager 155. Alert manager 155 can be in communication with wearanalyzer 153 and may monitor the wear trend and wear level determined bywear analyzer 153. Alert manager 155 can provide messaging alerts tooperator control panel 130 based on information determined by wearanalyzer 153. For example, when the wear level reaches a wear thresholdvalue, alert manager 155 may generate an alert that is shown on display133 of operator control panel 130. The threshold value can correspond tovalues indicating extreme GET wear or, in some cases, complete GET loss.The alert may provide an indication to the operator of work machine 100that one or more GET 125 need replacement. The wear threshold value canvary from embodiments and may dependent on the type of GET 125 and thematerial at the worksite with which GET 125 engage.

Alert manager 155 can also provide an alert that GET 125 may needreplacement at some point in the future, for example, that GET 125 mayneed to be replaced within two weeks. A replacement alert can includeinformation related to wear trend predictions for GET 125. For example,the replacement alert can include a quantification of the wear trend(e.g., GET 125 wear 2% per workday), the amount of time the teeth havebeen in use, or the expected date or time GET 125 will reach the wearthreshold based on usage data.

In some embodiments, alert manager 155 can monitor the wear trenddetermined by wear analyzer 153 and provide a wear level value todisplay 133 to inform operator of work machine 100 of the current wearlevel. For example, if the wear trend indicates that GET 125 are 60%worn down, based on the wear trend, alert manager 155 may provide anindication that GET 125 have 40% of their life left before they need tobe replaced. The display 133 can also inform an operator that a toothhas broken, indicating tooth loss (e.g., when one or more of GET 125have less than 20% life).

In some embodiments, alert manager 155 can generate instructions thatcause wear levels to be rendered on display 133 showing a wear level ormeasurement of GET 125. For example, if wear analyzer 153 determines,based on processed imaging data, that one of GET 125 is currently 325mm, alert manager 155 may generate instructions that when executed byprocessor(s) cause display 133 to show that the one GET is currentlymeasures 325 mm.

Wear detection computer system 110 can be in communication with a remotemonitoring computer system 160 in some implementations. Remotemonitoring computer system 160 can include a remote display 163, one ormore remote processor(s) 165, and a remote memory 170. Remote monitoringcomputer system 160 can be located at the work site where one or morework machines 100 operate and may be in communication with associatedinstances of wear detection computer system 110 at the work site. Remotemonitoring computer system 160 can be configured to display GET wearlevels, and store GET wear information, for multiple work machines 100to facilitate monitoring of GET health throughout a work site. Forexample, remote display 163 may be configured to show user interfacescorresponding to the one or more work machines 100 to display respectiveGET 125 wear levels in one location for ease of monitoring. Remotemonitoring computer system 160 may be implemented as a laptop computer,a desktop computer system, or a mobile device.

Remote monitoring computer system 160 includes a one or more remoteprocessors 165. Remote processor(s) 165 can include one or more of acentral processing unit (CPU), a graphics processing unit (GPU), afield-programmable gate array (FPGA), some combination of CPU, GPU, orFPGA, or any other type of processing unit. Remote processor(s) 165 mayhave numerous arithmetic logic units (ALUs) that perform arithmetic andlogical operations, as well as one or more control units (CUs) thatextract instructions and stored content from processor cache memory, andthen executes the instructions by calling on the ALUs, as necessary,during program execution. Remote processor(s) 165 may also beresponsible for executing drivers and other computer-executableinstructions for applications, routines, or processes stored in remotememory 170, which can be associated with common types of volatile (RAM)and/or nonvolatile (ROM) memory.

Remote monitoring computer system 160 also includes remote memory 170.Remote memory 170 can include system memory, which may be volatile (suchas RAM), non-volatile (such as ROM, flash memory, etc.) or somecombination of the two. Remote memory 170 can further includenon-transitory computer-readable media, such as volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information, such as computer-readableinstructions, data structures, program modules, or other data. Systemmemory, removable storage, and non-removable storage are all examples ofnon-transitory computer-readable media. Examples of non-transitorycomputer-readable media include, but are not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other non-transitory medium which can be used to store thedesired information and which can be accessed by remote monitoringcomputer system.

Remote memory 170 stores data, including computer-executableinstructions, for remote monitoring computer system 160 to command andcontrol remote display 163 and to implement remote log manager 175.Remote memory 170 can also store additional components, modules, orother code executable by remote processor(s) 165 to enable operation ofremote monitoring computer system 160. For example, remote memory 170can include code related to input/output functions, software drivers,operating systems, or other components.

Remote monitoring computer system 160 includes a remote display 163which produces monitoring output for a manager of a work site that isrelated to the status of GET health and to receive alerts and alarmsrelated to GET wear levels for one or more work machines 100 at a worksite. Remote display 163 can include a liquid crystal display (LCD), alight emitting diode display (LED), cathode ray tube (CRT) display, orother type of display known in the art. In some examples, remote display163 includes audio output such as speakers or ports for headphones orperipheral speakers. Remote display 133 can also include audio inputdevices such as microphone or ports for peripheral microphones. Remotedisplay 133 includes a touch-sensitive display screen in someembodiments, which also acts as an input device.

Like display 133 (on work machine 100), remote display 163 of remotemonitoring computer system 160 can display information about the wearlevel or loss of GET 125 that has been rendered by wear computerdetection system 110 at a work site. For example, remote display 163 maydisplay calculated measurements of GET 125 for one or more work machines100. The calculated measurements may be color coded in some embodimentsto reflect a health status of GET 125. For example, the calculatedmeasurements may be displayed with a green background if GET 125 isconsidered to have an acceptable wear level, yellow background if GET125 are close to needing replacement, or red if GET 125 have broken orworn to the point of needing replacement. Display 133 can also show animage of bucket 120, GET 125, or an image of a region of interest withinfields-of-view 127, 129 related to GET 125 rendered by respective wearcomputer detection systems 110 of work machines 100 at a work site.

Wear detection computer system 110 allows an operator of work machine100 to be informed when GET 125 need replacement, or has broken, due toextensive wear. The processes employed by wear detection computer system110—which are described in more detail below—provide for accurate andprecise measurement of GET wear at a scale of less than 5 mm allowing anoperator to halt operation of work machine 100 in the event of extremeGET wear or loss. The processes and techniques deployed by weardetection computer system 110 can be used with a variety of workmachines.

For example, FIG. 2 is a diagram depicting a schematic side view of anexample environment 200 in which a wheel loader work machine 201 isoperating. Wheel loader work machine 201 can include a bucket 220 andone or more GET 225. As shown in FIG. 2 , a sensor 226 and a camera 228are positioned so that GET 225 and bucket 220 are within a field-of-view227 (of sensor 226) and field-of-view 229 (of camera 228) during a dumpend of the dig-dump cycle. As a result, LiDAR sensor 226 and camera 228can be configured in such embodiments to capture imaging data whenbucket 220 is at rest at the dump end of the dig-dump cycle.

As another example, FIG. 3 is a diagram depicting a schematic side viewof an example environment 300 in which a hydraulic mining shovel workmachine 301 is operating. Hydraulic mining shovel work machine 301 caninclude a bucket 320 and one or more GET 325. In contrast to thepositions of sensor 226 and camera 228 for wheel loader work machine201, a sensor 326 and a camera 328 are positioned such that GET 325 arewithin a field-of-view 327 (of sensor 326) and field-of-view 329 (ofcamera 328) during a dig end of the dig-dump cycle. Sensor 326 andcamera 328 can be configured in such embodiments to capture imaging datawhen bucket 320 is at rest at the dig end of the dig-dump cycle.

In yet another example, FIG. 4 is a diagram depicting a schematic sideview of example an environment 400 in which an electric rope shovel workmachine 401 is operating. Electric rope shovel work machine 401 caninclude a bucket 420, one or more GET 425, a sensor 426 and a camera428. As shown in FIG. 4 , GET 425 may be within a field-of-view 427 (ofsensor 426) and field-of-view 429 (of camera 428) at a midpoint in thedig-dump cycle, but when bucket 420 is relatively close to sensor 426and camera 428. In such embodiments, sensor 426 and camera 428 can beconfigured to capture imaging data when bucket 420 enters a range ofpositions correlating to field-of-view 427 and field of view 429.

It should be noted that FIGS. 2-4 are merely examples for particularwork machines and sensor/camera locations, but sensors 226, 326, 426 andcameras 228, 328, 428 can be positioned so that their respectivefields-of-view 227, 327, 427, 229, 329, 429 capture image data at anypoint in the dig-dump cycle of work machines 201, 301, 401. Moreover,the described positioning of sensors 226, 326, 426 and cameras 228, 328,428 can be combined in some embodiments. For example, the presentdisclosure contemplates embodiments of wheel loader work machine 201,hydraulic mining shovel work machine 301, and electric rope shovel workmachine 401 having sensors and cameras with fields-of-view directed tothe beginning, middle, and/or end of the dig-dump cycle.

FIG. 5 depicts an image data flow diagram 500 depicting an example flowof imaging data for a region of interest detection process usingcomputer vision techniques. Image data flow diagram 500 includes imagesthat are received, processed, and generated by image analyzer 150 whendetecting regions of interest within imaging data captured by camera 128related to GET 125. Image data flow diagram 500 includes a left image510 and a right image 520 captured by camera 128. Left image 510 can bea rectified image captured by the left image sensor of camera 128. Rightimage 520 can be rectified image captured by the right image sensor ofcamera 128. Both left image 510 and right image 520 include images ofbucket 120 and GET 125.

Image analyzer 150 may process left image 510 and right image 520 tocreate disparity map 530. Disparity map 530 can be a dense stereodisparity map showing the disparity between each pixel of left image 510and each pixel of right image 520. Using disparity map 530 and aphysical parameter set 535, obtained from physical parameter library 145and associated with bucket 120, GET 125 and/or work machine 100, imageanalyzer 150 can build a three-dimensional point cloud 540. 3D pointcloud 540 shows disparity between left image 510 and right image 520 inthree dimensions. Image analyzer 150 may then perform a segmentationanalysis on three-dimensional point cloud 540 to identify a region ofinterest 550 including GET 125 within left image 510, right image 520,or both. In some embodiments, image analyzer 150 may use region ofinterest 550 to command and control sensor 126 to capturehigher-resolution imaging data for GET 125.

FIG. 6 depicts an image data flow diagram 600 depicting an example flowof imaging data for a region of interest detection process using deeplearning techniques. Similar to image data flow diagram 500 describedabove, the output of the region of interest detection process will be aregion of interest 550 corresponding to GET 125 that image analyzer 150will then use to further analyze the image data. But unlike image dataflow diagram 500, image analyzer 150 utilizes deep learning techniquesto detect region of interest 550 in image data flow diagram 600.

Image data flow diagram 600 includes image 610 captured by camera 128.Image 610 could be a rectified image captured by either the left imagesensor or the right image sensor of camera 128, or it could be imagedata captured from sensor 126. Image analyzer 150 may apply a deeplearning GET detection algorithm to image 610. The deep learning GETdetection algorithm may employ a neural network that has been trainedwith a corpus of image data where GET have been individually identifiedand labeled and/or groups of GET have been individually identified andlabeled. When image analyzer 150 applies the deep learning GET detectionalgorithm to image 610, it may identify a plurality of individual GETbounding boxes 620 containing images of individual GET 125. In someembodiments, image analyzer 150 may also identify a GET group boundingbox 630 encompassing individual GET bounding boxes 620. Once imageanalyzer 150 identifies GET group bounding box 630 it may extract thepixels within it as region of interest 550. It is noted that while FIG.6 shows detection of region of interest 550 from image 610, which mayrepresent image data from the left image sensor of camera 128, the rightimage sensor of camera 128, the color image sensor of camera 128, orimage data captured by sensor 126, in some embodiments, the image dataflow diagram 600 may apply to more than one of these for a particulartime instance. For example, image analyzer 150 may detect a region ofinterest in imaging data from more than one of the left image sensor ofcamera 128, the right image sensor of camera 128, the color image sensorof camera 128, and/or image data captured by sensor 126 at a particulartime.

FIG. 7 depicts an image data flow diagram 700 depicting an example flowof imaging data for a region of interest detection process using deeplearning techniques. Similar to image data flow diagram 600 describedabove, the output of the region of interest detection process will be aregion of interest 550 corresponding to GET 125 that image analyzer 150will then use to further analyze image data. But unlike image data flowdiagram 500, the deep learning GET detection algorithm described withrespect to FIG. 7 has been trained using a corpus of data includingdisparity maps and regions of interest are detected using a disparitymap generated from more than one image sensor (e.g., the left imagesensor and right image sensor of camera 128).

Image data flow diagram 700 includes a first image 710 and a secondimage 720 captured by camera 128 or sensor 126. As one just example,first image 710 can be a rectified image captured by the left imagesensor of camera 128 and second image 720 can be rectified imagecaptured by the right image sensor of camera 128, but first image 710and second image 720 need not necessarily be received from camera 128.As another example, one of first image 710 or second image 720 could bean infrared image data captured from sensor 126. As yet another example,first image 710 could be captured by sensor 126 while second image 720could be captured by camera 128, or vice versa. Both first image 710 andsecond image 720 include images of bucket 120 and GET 125. Imageanalyzer 150 may process first image 710 and second image 720 to createdisparity map 730. Disparity map 730 can be a dense stereo disparity mapshowing the disparity between each pixel of first image 710 and eachpixel of second image 720.

Image analyzer 150 may apply a deep learning GET detection algorithm todisparity map 730. The deep learning GET detection algorithm may employa neural network that has been trained with a corpus of image data whereGET have been individually identified and labeled and/or groups of GEThave been individually identified and labeled within disparity mapscorresponding to the type of bucket 120 and GET 125 captured withinfirst image 710 and second image 720. When image analyzer 150 appliesthe deep learning GET detection algorithm to disparity map 730, it mayidentify a plurality of individual GET bounding boxes 740 containingimages of individual GET 125. In some embodiments, image analyzer 150may also identify a GET group bounding box 750 encompassing individualGET bounding boxes 740. Once image analyzer 150 identifies GET groupbounding box 750 it may extract the pixels within it as region ofinterest 550.

FIG. 8 depicts an image data flow diagram 800 depicting an example flowof imaging data for a region of interest detection process for imagingdata captured by sensor 126. The description of FIG. 8 that follows usesa LiDAR embodiment of sensor 126 as an example, but other examples arecontemplated. The imaging data captured by sensor 126 according to imagedata flow diagram 800 substantially corresponds to the field of viewshown in image 810. As shown, the field of view includes bucket 120 andGET 125.

Sensor 126 performs a LiDAR data capture that includes a plurality ofLiDAR “hits” for when sensor 126 detects an object surface, e.g., asurface corresponding to either bucket 120 or GET 125. The LiDAR hitscan be represented as three-dimensional point cloud 820, where eachpoint of three-dimensional point cloud 820 corresponds to a LiDAR hit.Image analyzer 150 determines region of interest of 510 based onthree-dimensional point cloud 820 by performing a segmentation analysisor other object recognition analysis technique. In some embodiments,image analyzer 150 may use physical parameter set 535 to identify regionof interest 550. For example, image analyzer 150 may use a bucket-toothtemplate, CAD-based model of GET 125, or pattern matching techniques toidentify region of interest 550 within three-dimensional point cloud820.

FIG. 9 depicts an image data flow diagram 900 depicting an example flowof imaging data for a wear detection process using marker pointidentification. As noted above, features of GET 125 can be consideredmarker points, and wear detection computer system 110 may determine GET125 measurements based on identification of marker points within imagedata, or regions of interest in the image data corresponding to GET 125.Marker points can include, for example, edges, corners or specificangles of GET 125. Data flow diagram 900 shows the flow of data formarker point identification using one region of interest (e.g., regionof interest 910), but in operation, wear detection computer system 110may process imaging data according to data flow diagram 900 for multiplesets of image data received from sensor 126 and/or camera 128 per timeinstance.

Once image analyzer 150 identifies region of interest 910, it referencesphysical parameter set 535 to assist in identifying marker points 920within region of interest 910 that correspond to GET 125. For example,physical parameter set 535 may include the size or shape of angles ofeach corner of GET 125 which image analyzer may use when performingcorner detection analysis on region of interest 910. As another example,physical parameter set 535 may include a template that matches markerpoints in GET 125 (e.g., an image of a corner, mark, weld, or otheridentifier) that image analyzer 150 applies in a segmentation analysisto region of interest 910. In embodiments where region of interest 910corresponds to image data captured by an IR camera, physical parameterset 535 may include temperature information related to expectedtemperature values of GET 125 and the work environment in which GET 125are used.

Once image analyzer 150 determines marker points 920, it can identifyand/or otherwise determine GET ends 930. GET ends 930 correlate to thelength of GET 125, with one end correlating to the bucket-side edge ofGET 125 (e.g., where GET 125 meets bucket 120) and another endcorrelating to the front or engaging edge of GET 125 (e.g., where GET125 engages with ground). Based on the determination of GET ends 930,wear analyzer 153 can determine GET measurements 940 for GET 125 thatare within region of interest 910. For example, wear analyzer 153 maydetermine the number of pixels between GET ends 930 for a particular GET125, then convert the pixel count to a distance measurement (e.g., areal-world distance measurement in metric or imperial units).

FIG. 10 shows a flowchart representing an example wear detection process1000 to detect wear of GET 125. In some embodiments, process 1000 can beperformed by image analyzer 150 and wear analyzer 153. Process 1000generally follows the image data flows of FIGS. 5-9 and should beinterpreted consistent with the description of these figures, and thedescriptions of image analyzer 150 and wear analyzer 153 described abovewith respect to FIG. 1 . Although the following discussion describesaspects of process 1000 being performed by image analyzer 150 or wearanalyzer 153, other components of wear detection computer system 110 mayperform one or more blocks of process 1000 without departing from thespirit and scope of the present disclosure.

Process 1000 begins at step 1010 where image analyzer 150 receivesimaging data from a plurality of sensors associated with work machine100. The plurality of sensors can include sensor 126 and the leftmonochrome, right monochrome, or color image sensors of camera 128, forexample. In some embodiments, each of the plurality of sensors havedifferent fields of view. For example, the left monochrome image sensorof camera 128 and the right monochrome image sensor of camera 128 mayhave slightly different fields of view to create the parallax needed fora stereo imaging. As another example, sensor 126 and camera 128 may havediffering orientations on work machine 100 to capture image data relatedGET 125 from different angles. Image analyzer 150 receives imaging datafrom each of the plurality of sensors during a dig-dump cycle of thework machine, and image analyzer 150 may correlate the respectiveimaging data for the plurality of sensors for analysis purposes. Forexample, for one iteration of process 1000, image analyzer 150 mayreceive first image data from one of the plurality of sensors, andsecond image data from another of the plurality of sensors, and processthem together to determine a GET wear measurement for one dig-dumpcycle.

At step 1020, image analyzer 150 identifies respective regions ofinterest within the image data it received from the plurality of sensorsat step 1010. For example, image analyzer 150 may use computer visiontechniques (see e.g., FIG. 5 ) to identify a first region of interestwithin first image data received from a first of the plurality ofsensors and a second region of interest within second image datareceived from a second of the plurality of sensors. Image analyzer 150may also use deep learning techniques (see e.g., FIGS. 6, 7 ) toidentify a first region of interest within first image data receivedfrom a first of the plurality of sensors and a second region of interestwithin second image data received from a second of the plurality ofsensors. When one of the plurality of sensors is a LiDAR sensor, imageanalyzer 150 may use a point cloud analysis method to determine a regionof interest within image data (see e.g., FIG. 8 ).

At step 1030, image analyzer 150 may also identify one or more imagepoints within the regions of interest (see e.g., FIG. 9 ). In someexamples, the image points identified by image analyzer 150 at step 1030are associated with edges of GET 125 as described above with respect toFIGS. 1 and 9 . In some embodiments, image analyzer 150 uses geometricparameters describing GET 125 to determine the image points. Thegeometric parameters may describe the corners of GET 125 (e.g., relativelength of front edges to side edges, corner angles, corner shapes) orthey may describe other physical aspects of GET 125 such as overallsize, shape, or thickness.

At step 1040, wear analyzer 153 determines GET measurements based on theimage points identified at step 1030. Wear analyzer 153 may correlatepixel counts between image points to distance measurements (see e.g.,FIG. 9 ) to determine GET measurements. In some embodiments, wearanalyzer 153 may track and record GET measurements in pixels. Based onGET measurements determined at step 1040, wear analyzer 153 determineswear level or loss of GET at step 1050. The wear level or loss may bequantified in real-world measurements (e.g., millimeters), in terms ofpixels, or as a percentage of expected size (based, for example, on theCAD-based model for GET 125). As discussed above, wear analyzer 153 mayuse a CAD-based model of GET 125 in an unworn state and compare it tothe observed GET 125 measurement to determine GET wear level or loss.Wear analyzer 153 can also use historical measurement data for GET todetermine wear level over time or to determine a wear level trend tomake a prediction of when GET 125 will need replacement. In someembodiments, wear analyzer 153 may be configured to determine loss whenwear exceeds a threshold. For example, wear analyzer may determine lossof a GET if its size is more then 50% reduced, or reduced by a fixedmeasurement amount (e.g., 5 cm in length). Wear analyzer 153 maygenerate an alert when wear of GET meets or exceeds the threshold.

In some embodiments, image analyzer 150 and wear analyzer 153 performprocess 1000 several times within one dig-dump cycle. In suchembodiments, wear analyzer 153 may compare GET measurements determined(step 1040) at the current point in the dig-dump cycle to one or moreGET measurements determined earlier (or later) within the same dig-dumpcylce, or to historical measurements and determine whether the currentlydetermined GET measurement is consistent. If the measurement isinconsistent—e.g., it differs from other GET measurements within thesame dig-dump cycle by some threshold value—wear analyzer 153 candiscard the current measurement as noise or erroneous data. Thethreshold value can be configured based on the environment, workmachine, or type of GET. For example, for a work machine digging softermaterials with long GET, the threshold may be set to a low value (e.g.,less than 10%) and for a work machine digging hard materials withshorter GET, the threshold may be set to a higher value (e.g., greaterthan 20%). In addition, in some embodiments, image analyzer 150 and wearanalyzer 153 may also consider the point in time within the dig-dumpcycle in when it performs process 1000 to determine whether a determinedGET measurement is noise.

FIG. 11 shows a flowchart representing an example wear detection process1100 to detect wear of GET 125. In some embodiments, process 1100 can beperformed by image analyzer 150 and wear analyzer 153. Process 1100generally follows the image data flows of FIGS. 5-9 and should beinterpreted consistent with the description of these figures, thedescriptions of image analyzer 150 and wear analyzer 153 described abovewith respect to FIG. 1 , and wear detection process 1000. Although thefollowing discussion describes aspects of process 1100 being performedby image analyzer 150 or wear analyzer 153, other components of weardetection computer system 110 may perform one or more blocks of process1100 without departing from the spirit and scope of the presentdisclosure.

Process 1100 can be performed within one dig-dump cycle of work machine100 or within/across multiple dig-dump cycles of work machine 100. Insome embodiments, image analyzer 150 and wear analyzer 153 performprocess 1100 several times within a single dig-dump cycle.

After the start of the dig-dump cycle (block 1105), at step 1110 imageanalyzer 150 receives first image data from sensor 126 or camera 128.After processing the image first image data (using the process 1000 asjust one non-limiting example), wear analyzer 153 determines a firstwear measurement for GET 125 at block 1115 and from that determines afirst wear level for GET 125. Wear analyzer 153 may determine the firstwear measurement and first wear level using techniques described abovewith respect to FIGS. 1, 9 and 10 as just some examples.

After wear analyzer 153 determines the first wear level for GET 125, itwill determine whether the first wear level is indicative of a GETreplacement condition, e.g., if wear analyzer 153 should generate analert to notify the operator of work machine 100 that a GET needsreplacement. The determination of whether the first wear level isindicative of a GET replacement condition can contain two decisions insome embodiments. One decision is whether the wear level indicates theneed for alert (step 1127). If the wear level does not indicate the needfor an alert (step 1127: NO), process 1100 ends and may be repeated atanother time within the same dig-dump cycle consistent with disclosedembodiments. (block 1195). If, however, the wear level indicates theneed for an alert (step 1127: YES), processing moves to the otherdecision (step 1129)—whether the first image data was captured late inthe dig-dump cycle. In some embodiments, the other decision may be basedon a “late point” within the dig-dump cycle. If the first image data isreceived after the late point, then wear analyzer 153 may determine thatit is late in the dig-dump cycle (step 1129: YES), process 1100 willproceed to step 1135 and wear analyzer 153 will generate an alert. Ifthe first image data is received before the late point, then wearanalyzer 153 may determine that it is not late in the dig-dump cycle(step 1129: NO), and process 1100 will proceed to step 1140. The latepoint can be set by the operator of work machine 100 in some embodimentsand the late point may have a default setting. An example defaultsetting is the mid-point or 50% point in the dig-dump cycle—image datacaptured closer to the end of the dig-dump cycle is captured late in thecycle whereas image data captured closer to the beginning of thedig-dump cycle is captured early (not late) in the dig-dump cycle.

When wear analyzer 153 determines that the first GET wear levelindicates the need for an alert and it did not receive the first imagedata late in the dig-dump cycle, a second set of image data for GET 125will be collected to confirm the need for an alert. At step 1140, imageanalyzer 150 receives second image data from sensor 126 or camera 128.After processing the second image data (using the process 1000 as justone non-limiting example), wear analyzer 153 determines a second wearmeasurement for GET 125 at block 1145 and from that determines a secondwear level for GET 125 (step 1150) in way similar to how it determinedthe first wear level for GET 125, described above. Wear analyzer 153then determines whether the GET replacement condition has beensatisfied. If both the first wear level and the second wear levelindicate the need for an alert (step 1155: YES), then wear analyzer willgenerate an alert at step 1135. If, however, the second wear level doesnot indicate the need for an alert (step 1155: NO), the GET replacementcondition is not satisfied and process 1100 ends for the currentdig-dump cycle (block 1195).

In some embodiments, wear analyzer 153 may log the first GET wear levelor second GET wear level (when calculated) for each iteration of process1100. Logging may include storing a the GET wear measurement, GET wearlevel, and/or storing captured image data. Wear analyzer 153 may logthis information local to wear detection computer system 110 in GET wearlevel storage 157, or provide it to remote monitoring computer systemfor storage in remote log manager 175.

FIG. 12 shows an example wear detection user interface 1200 that can berendered on display 133 or remote display 163. User interface 1200 caninclude a captured image 1210 of bucket 120 and GET 125. Captured image1210 can be a still image or real-time video image of bucket 120 and GET125 in some embodiments. User interface 1200 can also include statususer interface elements 1220 for GET 125 that display the current statusof GET 125. Status user interface elements 1220 can display, forexample, the current measurement of GET 125 (as shown in FIG. 12 ),percentage wear of GET 125, or both. In some embodiments, status userinterfaces may include indicia of GET wear such as color coding toprovide information to the operator of work machine 100. For example,when GET 125 are in good health without significant wear, status userinterface elements 1220 may render with a healthy indicator 1230.Healthy indicator 1230 can be color coded (e.g., green) to indicate noaction need be taken. As another example, when GET 125 are partiallyworn and close to needing replacement, status user interface elements1220 may render with a partial wear indicator 1240. Partial wearindicator 1240 can be color coded (e.g., yellow), in special font, orappear highlighted within user interface 1200. As another example, whenGET 125 are worn to the point of needing replacement, or have broken,status user interface elements 1220 may render with a complete wearindicator 1250. Complete wear indicator 1250 can be color coded (e.g.,red), in special font, appear highlighted or consistently flash withinuser interface 1200 to alert the operator.

User interface 1200 can also include a close-up view 1260 of GET 125.Close-up view 1260 may correspond, for example, to identified regions ofinterest within image data. In some embodiments, in addition to completewear indicator 1250, user interface 1200 can include warningnotification 1270 to alert the operator of a GET replacement condition.Audio cues or alarms may also accompany partial wear indicator 1240,complete wear indicator 1250, and/or warning notification 1270.

Throughout the above description, certain components of wear detectioncomputer system 110 were described to perform certain operations. But,in some embodiments of wear detection computer system 110, othercomponents may perform these operations other than what is describedabove. In addition, wear detection computer system 110 may includeadditional or fewer components than what is presented above in exampleembodiments. Those of skill in the art will appreciate that weardetection computer system 110 need not be limited to the specificembodiments disclosed above.

INDUSTRIAL APPLICABILITY

The systems and methods of this disclosure can be used in associationwith operation of work machines at a worksite that are excavating,moving, shaping, contouring, and/or removing material such as soil,rock, minerals, or the like. These work machines can be equipped with abucket used to scoop, dig, or dump the material while at the worksite.The bucket can be equipped with one or more GET to assist with theloosening of the material during operation. The work machines can alsoinclude a system having a processor and memory configured to performmethods of wear detection according to the examples described herein.The systems and methods can detect wear or loss of work machinecomponents such as GET so operators of such work machines can takecorrective action before a failure damaging downstream processingequipment can occur.

In some examples, the systems and methods capture imaging dataassociated with GET from one or more sensors of the work machine that isthen processed to determine wear or loss of the GET. The one or moresensors can include LiDAR sensors, image sensors, and/or stereoscopiccameras.

In some examples, the systems and methods use image data from aplurality of sensors to more accurately determine GET wear. The systemsand methods detect image points within the image data to correlate GETmeasurements across the plurality of image sensors. By using imagepoints, processing across differing types of sensors can be simplifiedwith accuracy. Moreover, using image points can reduce computationalcomplexity of GET wear measurement by reducing the amount of data needto determine a measurement. The process described in the presentdisclosure provides high-precision measurements of GET while stillproviding processing efficiency.

While aspects of the present disclosure have been particularly shown anddescribed with reference to the examples above, it will be understood bythose skilled in the art that various additional embodiments may becontemplated by the modification of the disclosed devices, systems, andmethods without departing from the spirit and scope of what isdisclosed. Such embodiments should be understood to fall within thescope of the present disclosure as determined based upon the claims andany equivalents thereof. cm What is claimed is:

1. A computer-implemented method, comprising: receiving a plurality ofimages from a plurality of sensors associated with a work machine, theplurality of images comprising images of at least one ground engagingtool (GET) of the work machine, wherein individual sensors of theplurality of sensors have respective fields-of-view different from othersensors of the plurality of sensors; identifying a first region ofinterest associated with the at least one GET and included in a firstimage of the plurality of images, the first region of interest includingfirst data characterizing the at least one GET; identifying a secondregion of interest associated with the at least one GET and included ina second image of the plurality of images, the second region of interestincluding second data characterizing the at least one GET; determining afirst set of image points for the at least one GET, the first set ofimage points being determined based at least in part on geometricparameters associated with the at least one GET and first edges of theat least one GET detected in the first data, and determining a secondset of image points for the at least one GET, the second set of imagepoints being determined based at least in part on the geometricparameters associated with the at least one GET and second edges of theat least one GET detected in the second data, and determining a GETmeasurement for the at least one GET based at least in part on the firstset of image points and the second set of image points; and determininga wear level or loss for the at least one GET based on the GETmeasurement.
 2. The computer-implemented method of claim 1 wherein theplurality of sensors includes a left image sensor and a right imagesensor of a stereoscopic camera; and the plurality of images includes: aleft image of the at least one GET captured by the left image sensor,and a right image of the at least one GET captured by the right imagesensor.
 3. The computer-implemented method of claim 2 whereinidentifying the first region of interest includes generating a disparitymap based on the left image and the right image.
 4. Thecomputer-implemented method of claim 1 wherein the plurality of sensorsincludes an infrared image sensor.
 5. The computer-implemented method ofclaim 1 wherein identifying the first region of interest includesapplying a deep learning GET detection algorithm to the first image oridentifying the second region of interest includes applying a deeplearning GET detection algorithm to the second image.
 6. Thecomputer-implemented method of claim 1 wherein determining the wearlevel or loss for the at least one GET is based at least in part on aCAD-based model for the at least one GET.
 7. The computer-implementedmethod of claim 1 wherein determining the wear level or loss for the atleast one GET is based at least in part on previously determined wearlevels for the at least one GET.
 8. The computer-implemented method ofclaim 1 wherein the first set of image points or the second set of imagepoints correspond to at least one of a bucket-side edge of the at leastone GET or a front edge of the at least one GET.
 9. Thecomputer-implemented method of claim 1 further comprising generating analert when the wear level or loss is indicative of a GET replacementcondition.
 10. A system, comprising: a plurality of sensors associatedwith a work machine, wherein individual sensors of the plurality ofsensors have respective fields-of-view different from other sensors ofthe plurality of sensors, the respective fields-of-view directed tocapture a plurality of images of at least one ground engaging tool (GET)of the work machine; one or more processors; and non-transitory computerreadable media storing executable instructions that when executed by theone or more processors cause the one or more processors to performoperations comprising: identifying a first region of interest associatedwith the at least one GET and included in a first image of the pluralityof images, the first region of interest including first datacharacterizing the at least one GET; identifying a second region ofinterest associated with the at least one GET and included in a secondimage of the plurality of images, the second region of interestincluding second data characterizing the at least one GET; determining afirst set of image points for the at least one GET, the first set ofimage points being determined based at least in part on geometricparameters associated with the at least one GET and first edges of theat least one GET detected in the first data; determining a second set ofimage points for the at least one GET, the second set of image pointsbeing determined based at least in part on the geometric parametersassociated with the at least one GET and second edges of the at leastone GET detected in the second data; determining a GET measurement forthe at least one GET based at least in part on the first set of imagepoints and the second set of image points; and determining a wear levelor loss for the at least one GET based on the GET measurement.
 11. Thesystem of claim 10 wherein the plurality of sensors includes a leftimage sensor and a right image sensor of a stereoscopic camera; and theplurality of images includes: a left image of the at least one GETcaptured by the left image sensor, and a right image of the at least oneGET captured by the right image sensor.
 12. The system of claim 11wherein identifying the first region of interest includes generating adense stereo disparity map based on the left image and the right image.13. The system of claim 10 wherein the plurality of sensors includes aninfrared image sensor.
 14. The system of claim 10 wherein identifyingthe first region of interest includes applying a deep learning GETdetection algorithm to the first image or identifying the second regionof interest includes applying a deep learning GET detection algorithm tothe second image.
 15. The system of claim 10 wherein determining thewear level or loss for the at least one GET is based at least in part ona CAD-based model for the at least one GET.
 16. The system of claim 10wherein determining the wear level or loss for the at least one GET isbased at least in part on previously determined wear levels for the atleast one GET.
 17. The system of claim 10 wherein the first set of imagepoints or the second set of image points correspond to at least one of abucket-side edge of the at least one GET or a front edge of the at leastone GET.
 18. The system of claim 10 further comprising generating analert when the wear level or loss is indicative of a GET replacementcondition.
 19. A work machine, comprising: a bucket comprising at leastone ground engaging tool (GET); a stereoscopic camera comprising: a leftimage sensor, and a right image sensor; an infrared sensor; one or moreprocessors; and non-transitory computer readable media storingexecutable instructions that when executed by the one or more processorscause the one or more processors to perform operations comprising:receiving a left image of the at least one GET captured by the leftimage sensor; receiving a right image of the at least one GET capturedby the right image sensor; generating a disparity map based on the leftimage and the right image; identifying a first region of interestassociated with the at least one GET based on the disparity map, thefirst region of interest including first data characterizing the atleast one GET; receiving infrared image data from the infrared sensor;identifying a second region of interest associated with the at least oneGET within the infrared image data, the second region of interestincluding second data characterizing the at least one GET; determine afirst set of image points for the at least one GET, the first set ofimage points based at least in part on geometric parameters associatedwith the at least one GET and first edges of the at least one GETdetected in the disparity map; determine a second set of image pointsfor the at least one GET, the second set of image points determinedbased at least in part on the geometric parameters associated with theat least one GET and second edges of the at least one GET detected inthe second data; determine a GET measurement for the at least one GETbased at least in part on the first set of image points and the secondset of image points; and determining a wear level or loss for the atleast one GET based on the GET measurement.
 20. The work machine ofclaim 19 wherein identifying the first region of interest includesapplying a deep learning GET detection algorithm to the disparity map.